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NILPOTENT ORBITS, NORMALITY, 
AND HAMILTONIAN GROUP ACTIONS 



Ranee Brylinski and Bertram Kostant 

Abstract. Let M be a G-covering of a nilpotent orbit in g where G is a complex 
semisimple Lie group and g = Lie(G). We prove that under Poisson bracket the space 
R[2] of homogeneous functions on M of degree 2 is the unique maximal semisimple 
Lie subalgebra of R = R(M) containing g. The action of g' ~ R,[2] exponentiates to 
an action of the corresponding Lie group G' on a G'-cover M 1 of a nilpotent orbit in 
g' such that M is open dense in M' . We determine all such pairs (g C g'). 



The theory of coadjoint orbits of Lie groups is central to a number of areas in 
mathematics. A list of such areas would include (1) group representation theory, 
(2) symmetry-related Hamiltonian mechanics and attendant physical theories, (3) 
symplectic geometry, (4) moment maps, and (5) geometric quantization. From 
many points of view the most interesting cases arise when the group G in question 
is semisimple. For semisimple G the most familar of the orbits are of semisimple 
elements. In that case the associated representation theory is pretty much under- 
stood (Borel-Weil-Bott and noncompact analogues, e.g., Zuckerman functors). The 
isotropy subgroups are reductive and the orbits are in one form or another related 
to flag manifolds and their natural generalizations. 

A totally different experience is encountered with nilpotent orbits of semisimple 
groups. Here the associated representation theory (unipotent representations) is 
poorly understood and there is a loss of reductivity of isotropy subgroups. To 
make matters worse (or really more interesting) orbits are no longer closed and 
there can be a failure of normality for orbit closures. In addition simple connectivity 
is generally gone but more seriously there may exist no invariant polarizations. 

The interest in nilpotent orbits of semisimple Lie groups has increased sharply 
over the last two decades. This perhaps may be attributed to the reoccuring experi- 
ence that sophisticated aspects of semisimple group theory often leads one to these 
orbits (e.g., the Springer correspondence with representations of the Weyl group). 

In this note we announce new results concerning the symplectic and algebraic 
geometry of the nilpotent orbits O and the symmetry groups of that geometry. The 
starting point is the recognition (made also by others) that the ring R of regular 
functions on any G-cover M of O is not only a Poisson algebra (the case for any 
coadjoint orbit) but that R is also naturally graded. The key theme is that the 
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same nilpotcnt orbit may be "shared" by more than one simple group, and the 
key result is the determination of all pairs of simple Lie groups having a shared 
nilpotent orbit. Furthermore there is then a unique maximal such group and this 
group is encoded in the symplectic and algebraic geometry of the orbit. Remarkably 
a covering of nilpotent orbit of a classical group may "see" an exceptional Lie group 
as the maximal symmetry group of this sympectic manifold. A beautiful instance 
of this is that G2 is the symmetry group of the simply connected covering of the 
maximal nilpotcnt orbit of SL(3,C) and that this six-dimensional space "becomes" 
the minimal nilpotent orbit of G2 ■ 

Our work began with a desire to thoroughly investigate a striking discovery of 
Levasseur, Smith, and Vogan. They found that the failure of the closure of the 
eight-dimensional nilpotent orbit of G2 to be a normal variety may be "remedied" 
by refinding this orbit as the minimal nilpotent orbit of SO(7, C). The failure has 
a lot to do with the seven-dimensional representation of G2. In general given M 
we have found that there exists a unique minimal representation n (containing the 
adjoint) wherein M may be embedded with normal closure. It was the study of tt 
that led to the discovery of the maximal symmetry group G' . Using a new general 
transitivity result for coadjoint orbits we prove that, modulo a possible normal 
Heisenberg subgroup (and that occurs in only one case), G' is semisimple. 

Past experience has shown that the action of a subgroup H on a coadjoint orbit 
of G is a strong prognostigator as to how the corresponding representation L of G 
decomposes under H . If this continues to hold for unipotent representations our 
classification result should yield all cases where L remains irreducible (or decom- 
poses finitely) under a semisimple subgroup. 

1. THE MAXIMAL SYMMETRY GROUP AND "SHARED" ORBITS 

Let G be a simply connected complex semisimple Lie group and g the Lie algebra 
of G. Let e e g be nilpotcnt and assume (for simplicity of exposition but with no 
real loss) that e has nonzero projection in every simple component of g. Let O be 
the adjoint orbit of e and let v: M — ► O be a G-covering. Let R = R(M) be 
the ring of regular functions on M and let R[q] be the copy of g in R defining the 
moment map M — ► g*. Identify g* ~ g in a G-cquivariant way. 

Now R carries a G-invariant ring grading R = © fc>0 R[k] (k 6 Z) such that 
R[g] CR[2]. Theni?[0]=C. The Poisson bracket satisfies [R[k], R[l}} c R[k + l-2], 
for all k, I (see also [7]). Hence R[2] is a finite-dimensional Lie subalgebra of R under 
Poisson bracket. Our first main result is 

Theorem 1. 

(i) R[2] is a semisimple Lie algebra, call it g'. If g is simple then g' is simple . 

(ii) The condition i?[g] C R[2] determines the G-invariant ring grading on R 
uniquely. 

(iii) R[2] + R[l] + R[0] is the unique maximal finite- dimensional Lie subalgebra 
of R containing R[g]. 

In Table 1 all the Lie algebras are complex simple; particularly, in (2) and (3) 
n > 2 and in (6) n > 3. The last column V is a representation of g, written as 
a sum of its irreducible components (only fundamental dominant representations 
occur here). In (3) V 8 L = f\ 2 C 2n where L ~ C is sp(2n)-invariant. In (8) V is 
the sum of the standard and the two half-spin representations. 
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Table 1 



V 



(1) 
(2) 
(3) 
(4) 
(5) 
(6) 
(7) 
(8) 
(9) 



G 2 

so(2n + 1) 



50(7) 

0o(2n + 2) 
sl(2n) 




sp(2n) 

F A 

S [(3) 

so(2n) 

so(9) 

so(8) 

G 2 



G 2 



(A 2 C 2n )/C 

C 26 

C 3 © A 2 C 3 



Fi 

80(8) 



so(2n+ 1) 



C 2n 
C 16 



c 8 © c 8 © c 8 

2C 7 



Theorem 2. Table 1 gives a complete list of the simple Lie algebra pairs (g C g') 
that arise in Theorem 1 with g ^ g' . Furthermore g' = g © V as g-modules. 

Now the Poisson bracket on R defines an alternating bilinear form [3 on R[l] and 
a Lie algebra homomorphism R[2] — > sp /3. 

Theorem 3. (3 is a symplectic form on R[l] so that R[l] © R[0] is a Heisenberg 
Lie algebra. If g is simple then R[l] ^ in one and only one case, namely, when 
g is of type C n for some n and M is the simply connected (double covering) of the 
minimal nontrivial nilpotent orbit of g. In that case R[2] ~ sp(2n) and R[l] ~ C 2 ™ 
generates R freely. 

Now the functions in g' ~ R[2] define a map 4>: M — > g' (again identify (g')* ~ 
g' ). Let G' be the simply connected Lie group with Lie algebra g 1 . The next result 
says that up to birationality cf) is a moment map for G' . 

Theorem 4. The image <j>(M) lies in a nilpotent orbit O' of G' and 4>(M) is 
Zariski open dense in O' . There exists a unique G 1 -covering v' : M' — ► O' such 
that M' contains M and v' extends <j). Moreover M and M' have the same regular 
functions, that is, R(M) — R(M'), and also the same fundamental groups, that is, 
m(M) ~7ri(M'). 

We can construct M' in the following way. Given M, let X = Speci? be the 
maximal ideal spectrum of the finitely generated C-algebra R. Then X is a normal 
affine variety. Furthermore X contains M as an open dense subset. We call X the 
normal closure of M. Indeed if M = O and O is normal then X = O. 

Our construction is: G' acts on X and M' is the unique Zariski open orbit of 
G' on X. Note that M is the unique Zariski open orbit of G on X so that clearly 
M CM' CX. 

Thus the pair (M C M') constitutes an orbit (cover) "shared" between G and 
G' . Moreover M and M' have the same normal closure so that X is exactly shared 
between G and G' . 

Now even though X may be singular we will say an isomorphism of X is symplec- 
tic if the corresponding automorphism of R preserves the Poisson bracket structure. 

Theorem 5. Assume R[l] = 0. Then X is a singular variety. The action of any 
connected Lie group of holomorphic symplectic isomorphisms of X that extends the 
action of G is given by a subgroup of G' . 
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Hence, assuming g is simple, the example of Theorem 3 is the one and only one 
choice of M such that X is smooth, and in that case one has X ~ C 2 ™. 

2. Explanation of the table 

We now describe for each of the nine cases in Table 1 a choice of M such that 
(M, g) gives rise to (M',g'). In each case M' = O' is the orbit of the highest root 
vector in the simple Lie algebra g', this is the minimal nontrivial nilpotcnt orbit 
(cf. the next section). 

In cases (l)-(4) g is any one of the four simple Lie algebras that are doubly 
laced (i.e., having two root lengths). Choose O to be the orbit of a short root 
vector and let M be its simply connected cover. Then M = O in (1) while M is 
a two- fold cover of O in (2)-(4). Then g' is simply laced . Furthermore V = V a 
is the irreducible g-representation with highest weight a equal to the highest short 
root of g. Case (1) is a restatement in our language of a result proved by Levasseur 
and Smith [4] in answer to a conjecture of Vogan [6] (see introduction). 

In (5) choose O to be the six-dimensional (maximal) nilpotent orbit of all princi- 
pal nilpotent elements and let M be the three-fold simply connected covering space 
of O. This case was discovered by us in collaboration with Vogan. A noncommu- 
tative analog of this example is given in a result of Zahid [8] . 

In (6) choose O to be the nilpotent orbit of Jordan type (see e.g., [3]) (3, 1 2 ™~ 3 ) 
and let M be the simply connected double cover of O. In (7) choose O to be of 
Jordan type (2 4 , 1) and let M be the simply connected (double cover) of O. In 
(8) choose O to be of Jordan type (3, 2 2 ,1) and let M be the simply connected 
(four-fold) cover of O. 

In (9) choose O to be the unique ten-dimensional nilpotent orbit and let M be 
the simply connected six-fold cover of O {-k\{0) — S3). In this example Levasseur 
and Smith already showed in [4] that G has an open dense orbit on O' , again in 
response to a question of Vogan. Moreover Vogan has constructed a unipotent 
representation theoretic analogue of the example. If tt denotes the minimal unitary 
representation of SO(4, 4) (see e.g., [2]) then w extends to the outer automorphism 
group A of g and in particular to a group S ~ S3 that induces A. Vogan shows 
that a split form G of G 2 and S behave like a Howe pair with respect to ir and 
that ir\G decomposes into six irreducible components. Furthermore McGovern in 
[5, Theorem 4.1] has constructed a Dixmier algebra analogue of this example 

Remarkably three of the four nilpotent orbits of G2 have now appeared as 
"shared" orbits (the principal orbit does not appear). 

Cases (5)-(8) are precisely the the pairs (g C g') in the table with rank g 
=rank g'. Each pair (g C g') is of the form (s C s) where s is a doubly laced 
simple Lie algebra and So is a subalgebra of s containing a Cartan subalgebra of 
s and all associated long root vectors. Moreover every such pair (so C s) where 
s is simple arises in (5)-(8). On the other hand the pairs (so C s) where So is 
nonsimple occur precisely when s ~ sp(2n) and So — sp(2rti) © • • • ©sp(2rifc) where 

n\ H V rik = n and k > 2. These pairs do arise in Theorem 1 (when M is chosen 

so that in each simple component one has the example of Theorem 3) and these 
together with (5)-(8) exhaust all equal rank pairs (g C g') arising in Theorem 1 
such that g' is simple. 

A general result regarding the ranks of g and g' is that rank g' > rank g whenever 
M = O and g ^ g'. 
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Two instances of a triple of Lie algebras having a "shared" orbit can be found 
among these examples, namely, (a) so(8) C so(9) C U (where f4 is of type F4) and 
(b) Q2 C so (7) C so (8). This is not unexpected by the theory since we in fact prove 
that if f) is any Lie subalgebra between g and g' then [) is semisimple and also t) is 
simple if g is simple. Moreover if H is the simply connected group corresponding 
to I) then the statements made in Theorem 4 and immediately afterward for G' and 
X apply equally well to H and X. In particular one has a unique open if -orbit 
in X, M C C M', and M*> covers H-equivariantly a nilpotcnt if -orbit O 1 ' C f). 
Furthermore the whole graded Poisson ring structure on R arising from M and O 
is the same as would arise from and . In particular the maximal semisimple 
Lie algebra R[2] remains the same. 

3. Methods of proof 

Two key ideas are used in proving the classification of pairs. The first is a 
representation theoretic. We are able to compute g' ~ R[2] as a jj-module (for 
arbitrary g and M). Let £ be a point of M lying over e and let (h,e,f) be a 
standard basis of an sl(2) subalgebra a of g. For any cj-module V let V[2] be the 
2-eigenspace of h in the fixed space (V*) GE . Then g' ~ g © rnV\ © • • • © n s V s where 
V\ , . . . , V s is a complete list of inequivalent simple cj-modules, excluding components 
of the adjoint representation, such that rn = dim V;[2] is nonzero. This follows by 
recognizing that the grading on R comes from exponentiating a natural action of a 
Cartan subalgebra in a . 

The second idea is due to David Vogan who observed that for a given pair (g,g r ) 
arising in Theorem 1 we may change (if necessary) the choice of O and M so that 
O' is minimal . Vogan himself has determined the pairs (g, g') in many of the cases 
listed above. 

A principle used in setting up the theory is that one should study minimal 
embeddings of X in order to study M (again M is arbitary). We prove the covering 
map v extends to a finite G-morphism v: X — > O and then the fiber j/ _1 (0) over 
zero is a single point o. Then o is the unique G-fixed point in X and the maximal 
ideal of R corresponding to o is m — ©^Li R[k). Furthermore, X is singular if 
and only if X is singular at o and then o is the "most" singular point of X (cf. 
Theorems 3 and 5). 

We say that the pair (v, V) defines an embedding of X in case V is a G-modulc 
and v <E V G (where e e M) is such that the natural map M — > G-v extends to a G- 
isomorphism X — > G • v. We find that the Zariski tangent space T D (X) — (m/m 2 )* 
at o provides a minimal embedding for X. 

Theorem 6. There exists a vector u G T Q (X) G ' such that (i) (u, T a (X)) defines 
an embedding of X and (ii) if (v, V) is any pair that defines an embedding of X 
then there exists a surjective G-map r: V — > T a (X) such that t(v) = u. 

Clearly one has an injection R[2] — ► m/m 2 when R[l] = 0. Regarding the 
normality of O we find that O is a normal variety if and only if as G-modules 
R[g] ~ R[2] ~ m/m 2 . 

The proofs of Theorems 1, 3, and 5 are all applications of the following general 
transitivity theorem for coadjoint orbits of a special kind of Lie algebra. This is 
one of our main results. 
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Theorem 7. Assume that s is a finite- dimensional Lie algebra over R or C and 
that s is a semidirect sum s = r + u where u is an abelian ideal in s, r is a semisimple 
Lie subalgebra of s, and u r = 0. One may regard s* = r* + u* in an obvious way. 
Let 7 G s* and write 7 = fi + A with (i£f and A e u* . T/ien one /ias s • 7 = r • 7 
z/ and on/y if X = 0. 

Theorem 7 says that if A ^ then the subgroup of Ads corresponding to r 
cannot operate transitively (even infinitesmally) at 7 on the coadjoint orbit. 

4. AN APPLICATION TO SYMMETRY OF FLAG VARIETIES 

Finally we give an application of our results to a well-known problem in geometry. 
If P is a parabolic Lie subgroup of a simple Lie group G it is a solved problem (see 
[1]) to determine the connected component of the full group F of holomorphic 
automorphisms of the projective variety G/P. It is precisely for the G given in 
cases (1), (2), and (3) in the table that P exists so that F is larger than that given 
by the action of G. Furthermore in those cases F is in fact given by the action of 
G'. We obtain a stronger statement (and recover the known result) in 

Theorem 8. Let P be any parabolic subgroup of G and choose M to be the unique 
open orbit in T*(G/P) so that we can take O to be the G-orbit in q of a Richard- 
son element in the nilradical of Lie P. Then one has a desingularization map 
T*(G/P) — > X and the pullback of R is the full ring of regular functions on 
T*(G/P). Furthermore the action of G' on X lifts uniquely to an action as a 
group of symplectic holomorphic automorphisms of T*(G/P) and as such G' is 
maximal. G' preserves the cotangent polarization of T* (G/P) so that G' acts on 
G/P and hence there exists a parabolic subgroup P' C G' such that G/P — G' /P'. 
The action of G' on G/P is the connected component of the group of all holomor- 
phic automorphisms of G/P. Consequently any connected Lie group of symplectic 
holomorphic automorphisms of T* (G/P) containing the action of G automatically 
preserves the cotangent space polarization of T* (G/P) and consequently will act as 
a group of holomorphic automorphisms of G/P. 
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